Expert knowledge elicitation to improve formal and mental models

Knowledge intensive processes are often driven and constrained by the mental models of experts acting as direct participants or managers. Descriptions of these relationships are not generally available from traditional data sources but are stored in the mental models of experts. Often the knowledge is not explicit but tacit, so it is dicult to describe, examine, and use. Consequently, improvement of complex processes is plagued by false starts, failures, institutional and interpersonal con ̄ict, and policy resistance. Modelers face diculties in eliciting and representing the knowledge of experts so that useful models can be developed. We describe and illustrate an elicitation method that uses formal modeling and three description format transformations to help experts explicate their tacit knowledge. We use the method to elicit detailed process knowledge describing the development of a new semiconductor chip. The method improved model accuracy and credibility and provided tools for development team mental model improvement. *c 1998 John Wiley & Sons, Ltd. Syst. Dyn. Rev. 14, 309±340, (1998) Many public and private sector systems increasingly depend on knowledge intensive processes managed and operated by interdisciplinary teams. These systems are dicult to manage. Often formal models such as system dynamics models are used to help managers understand the sources of diculties and design more e€ective policies. Typically, the expert knowledge of the people who actually operate the system is required to structure and parameterize a useful model. To develop a useful model that is also credible in the eyes of the managers, however, modelers must elicit from these experts information about system structure and governing policies, and then use this information to develop the model. While many methods to elicit information from experts have been developed, most assist in the early phases of modeling: problem articulation, boundary selection, identi®cation of variables, and qualitative causal mapping. These methods are often used in conceptual modeling, that is, in modeling e€orts that stop short of the development of a formal model that can be used to test hypotheses and proposed policies. The literature is comparatively silent, however, regarding methods to elicit the information required to estimate the parameters, initial conditions, and behavior relationships that must be speci®ed precisely in formal modeling. 309 Department of Information Sciences, University of Bergen, N-5020 Bergen, Norway. E-mail: David.Ford@ i®.uib.no Sloan School of Management, Massachusetts Institute of Technology, 50 Memorial Drive, E53-351 Cambridge, MA 02142, U.S.A. E-mail: [email protected] David N. Ford is an associate professor in the system dynamics program at the University of Bergen in Norway. He earned his PhD from the Massachusetts Institute of Technology, where he conducted research on the dynamics of development processes. His current research interests include product development management, coordination and policy development. John Sterman is the J. Spencer Standish Professor of Management at the Massachusetts Institute of Technology (MIT) Sloan School of Management, and Director of the MIT System Dynamics Group. Much of the information about system structure and decision processes resides in the mental models of process participants, where it remains tacit (Nonaka and Takeuchi 1995; Forrester 1994; Polanyi 1966). Compared to explicit knowledge, tacit knowledge is subjective, personal, and context-speci®c. It is dicult to describe, examine, and use. Therefore an important activity in modeling these systems is the elicitation, articulation, and description of knowledge held in the mental models of system experts. By system expert we mean those people who participate in the process directly in operational or managerial roles. We seek to improve modeling and mental model improvement techniques by proposing and testing a method of expert knowledge elicitation. The method we develop is designed to assist modelers and their clients specify parameters and relationships in a form suitable for formal modeling. We also argue, however, that the additional precision and discipline required to elicit information about these relationships in a form suitable for formal modeling can yield insights of value to modelers and clients even when no formal model is contemplated or built. We illustrate the use of the techniquewith an example drawn from a model of the development of a high-tech product. Product development is one of many processes in which globalization, accelerated technology evolution, and increased customer sophistication have resulted in a dramatic increase in complexity and a corresponding rise in cost overruns, delays, quality problems, and outright failures. Under pressure to bring new products to market ever faster and cheaper, methods such as concurrent development and co-located cross-functional teams have been widely adopted. Concurrent product development requires multiple knowledge-driven processes, such as design, which produces descriptions of the ®nal product and quality assurance and transforms unchecked designs into approved designs or designs requiring changes. E€ective product development and e€ective modeling of product development depend on knowledge of these critical process relationships, which are dynamic, nonlinear, biased by individual perspectives and goals, conditioned by experience, and aggregate many system components and relationships. Descriptions of process relationships are often not generally available from traditional data sources, such as company records, but are stored in the mental models of the process experts. Di€erences in the mental models of team members can constrain progress and lead to con ̄ict. The frequently divergent mental models of marketing managers and design engineers regarding the sequence of steps required to develop a product concept into a detailed design provide examples (Ford and Sterman 1998; Clark and Fujimoto 1990; Kim 1993). System dynamics models of these systems must include the process knowledge of system experts, which drives and constrains these processes (Barlas 1996; Williams et al. 1995; 310 System Dynamics Review Volume 14 Number 4 Winter 1998